Interoperability may be the ability of a computer system or application to work with other computer systems or applications, typically without special effort on the part of the users. Although many applications exist which can expose to another application or system an external graphical user interface (GUI) for interoperability via an application programming interface (API) or a software development kit (SDK) which allow interaction with the GUI by low-level programming techniques, there exist applications which cannot be connected or have their internal events easily known by any existing technique.
An example of an application which often cannot be connected by standard API and/or SDK techniques is an application executing or running in a remote environment (e.g., in a server/client architecture). In this case, the user (e.g. using the client system) may be able to see an image of an application on the screen of the client device and perform various input operations using a keyboard and/or mouse or touchscreen, but existing application integration techniques cannot recognize user interface (UI) elements or connect to any API exposed by the application, even when such APIs exist and are available or ready to use. As such, a third party application attempting to work with or access an application may not be able to. Examples of remote environments in which this problem is prevalent include the Microsoft Remote Desktop system, the Citrix XenApp system, the PCAnywhere system, and the Oracle VM system.
Interoperability problems can be attributed to one or more of for example lack of a reliable connector (e.g. API or SDK) for an interaction with such applications; and lack of an object model exposed by GUI elements or objects (for example, buttons, list boxes, links, tables, etc.).
One prior attempt at solving the interoperability issue includes technology focused on Optical Mark Recognition (OMR), which is used with Optical Character Recognition (OCR) engines to format text while generating a specific text document. OMR also provides the ability to recognize a text document with a template designed by developers. However, OMR and similar technologies do not provide an ability to interact with an application image, among other deficiencies.